10am Nov 24 started.....
mpiexec -np 4 -host localhost,node001,node002,node003 /common/clcbfxcell/blastall_cell_mpi -c cell_sw -v 1 -a 10 -b 1 -p blastx -m 8 -i /Volumes/bigfishRAID/Users/safs/Dropbox/Cluster/tmp/s_7_69686contigs.fa -d /common/clcbfxcell/databases/swissprot.fasta > /Volumes/bigfishRAID/Users/safs/Dropbox/Cluster/tmp/s_7_69686contigs_swiss.txt 


Ended Nov 25th 6 pm
@ contig 29778



ERROR

bigfish:~ safs$ mpiexec -np 4 -host localhost,node001,node002,node003 /common/clcbfxcell/blastall_cell_mpi -c cell_sw -v 1 -a 10 -b 1 -p blastx -m 8 -i /Volumes/bigfishRAID/Users/safs/Dropbox/Cluster/tmp/s_7_69686contigs.fa -d /common/clcbfxcell/databases/swissprot.fasta > /Volumes/bigfishRAID/Users/safs/Dropbox/Cluster/tmp/s_7_69686contigs_swiss.txt 

[node002:57332] *** Process received signal ***

[node002:57332] Signal: Segmentation fault (11)

[node002:57332] Signal code: Address not mapped (1)

[node002:57332] Failing at address: 0x0

[node002:57332] [ 0] 2   libSystem.B.dylib                   0x0000000080f803fa _sigtramp + 26

[node002:57332] [ 1] 3   blastall_cell_mpi                   0x00000000006ab5e8 _ZTS24CFixClockSecureStoreFile + 567019

[node002:57332] [ 2] 4   blastall_cell_mpi                   0x0000000000316313 _mh_execute_header + 3236627

[node002:57332] [ 3] 5   blastall_cell_mpi                   0x000000000030d1ee _mh_execute_header + 3199470

[node002:57332] [ 4] 6   blastall_cell_mpi                   0x000000000000ccad _mh_execute_header + 52397

[node002:57332] [ 5] 7   blastall_cell_mpi                   0x000000000000e7f1 _mh_execute_header + 59377

[node002:57332] [ 6] 8   blastall_cell_mpi                   0x000000000000ed0a _mh_execute_header + 60682

[node002:57332] [ 7] 9   blastall_cell_mpi                   0x0000000000006c3b _mh_execute_header + 27707

[node002:57332] [ 8] 10  blastall_cell_mpi                   0x0000000000008094 _mh_execute_header + 32916

[node002:57332] [ 9] 11  blastall_cell_mpi                   0x00000000002918d2 _mh_execute_header + 2693330

[node002:57332] [10] 12  blastall_cell_mpi                   0x0000000000001864 _mh_execute_header + 6244

[node002:57332] *** End of error message ***

[node003:79225] *** Process received signal ***

[node003:79225] Signal: Segmentation fault (11)

[node003:79225] Signal code: Address not mapped (1)

[node003:79225] Failing at address: 0x0

[node003:79225] [ 0] 2   libSystem.B.dylib                   0x00000000802843fa _sigtramp + 26

[node003:79225] [ 1] 3   blastall_cell_mpi                   0x00000000006ab5e8 _ZTS24CFixClockSecureStoreFile + 567019

[node003:79225] [ 2] 4   blastall_cell_mpi                   0x0000000000316313 _mh_execute_header + 3236627

[node003:79225] [ 3] 5   blastall_cell_mpi                   0x000000000030d1ee _mh_execute_header + 3199470

[node003:79225] [ 4] 6   blastall_cell_mpi                   0x000000000000ccad _mh_execute_header + 52397

[node003:79225] [ 5] 7   blastall_cell_mpi                   0x000000000000e7f1 _mh_execute_header + 59377

[node003:79225] [ 6] 8   blastall_cell_mpi                   0x000000000000ed0a _mh_execute_header + 60682

[node003:79225] [ 7] 9   blastall_cell_mpi                   0x0000000000006c3b _mh_execute_header + 27707

[node003:79225] [ 8] 10  blastall_cell_mpi                   0x0000000000008094 _mh_execute_header + 32916

[node003:79225] [ 9] 11  blastall_cell_mpi                   0x00000000002918d2 _mh_execute_header + 2693330

[node003:79225] [10] 12  blastall_cell_mpi                   0x0000000000001864 _mh_execute_header + 6244

[node003:79225] *** End of error message ***

[node001:32753] *** Process received signal ***

[node001:32753] Signal: Segmentation fault (11)

[node001:32753] Signal code: Address not mapped (1)

[node001:32753] Failing at address: 0x0

[node001:32753] [ 0] 2   libSystem.B.dylib                   0x0000000084e183fa _sigtramp + 26

[node001:32753] [ 1] 3   blastall_cell_mpi                   0x00000000006ab5e8 _ZTS24CFixClockSecureStoreFile + 567019

[node001:32753] [ 2] 4   blastall_cell_mpi                   0x0000000000316313 _mh_execute_header + 3236627

[node001:32753] [ 3] 5   blastall_cell_mpi                   0x000000000030d1ee _mh_execute_header + 3199470

[node001:32753] [ 4] 6   blastall_cell_mpi                   0x000000000000ccad _mh_execute_header + 52397

[node001:32753] [ 5] 7   blastall_cell_mpi                   0x000000000000e7f1 _mh_execute_header + 59377

[node001:32753] [ 6] 8   blastall_cell_mpi                   0x000000000000ed0a _mh_execute_header + 60682

[node001:32753] [ 7] 9   blastall_cell_mpi                   0x0000000000006c3b _mh_execute_header + 27707

[node001:32753] [ 8] 10  blastall_cell_mpi                   0x0000000000008094 _mh_execute_header + 32916

[node001:32753] [ 9] 11  blastall_cell_mpi                   0x00000000002918d2 _mh_execute_header + 2693330

[node001:32753] [10] 12  blastall_cell_mpi                   0x0000000000001864 _mh_execute_header + 6244

[node001:32753] *** End of error message ***

--------------------------------------------------------------------------

mpiexec noticed that process rank 2 with PID 57332 on node node002 exited on signal 11 (Segmentation fault).

--------------------------------------------------------------------------

bigfish:~ safs$ 



Restarted Nov 29 at 7am
stopped Nov 30 at 3pm

@ Contig29778